-
Notifications
You must be signed in to change notification settings - Fork 557
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
function to emit metrics #1649
function to emit metrics #1649
Conversation
53a5318
to
84cece7
Compare
7098fde
to
40bdb54
Compare
5a0d81d
to
56aa9d8
Compare
TODO new doc file |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can't speak to the technical implementation of this, but I think it needs docs.
I would suggest creating a docs/metrics.md
file to document why these exists, and the API for sending metrics. Maybe also some recommendations like "Use snake case for metric names" or "Put an x_ prefix on your custom metrics names" or similar.
docs/metrics.md
Outdated
@@ -0,0 +1,14 @@ | |||
# Metrics | |||
|
|||
Prediction objects have a `metrics` field. This normally includes `predict_time` and `total_time`. Official language models have metrics like `input_token_count`, `output_token_count`, `tokens_per_second`, and `time_to_first_token`. Currently, custom metrics from Cog are ignored when running on Replicate. Official Replicate-published models are the only exception to this. When running outside of Cog, you can emit custom metrics like this: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What does "When running outside of Cog" mean? Should this be "outside of Replicate"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed; we'll also have another chance to review this when merging into main
Signed-off-by: technillogue <technillogue@gmail.com>
Signed-off-by: technillogue <technillogue@gmail.com>
68a9891
to
0141cce
Compare
* function to emit metrics * add metrics docs --------- Signed-off-by: technillogue <technillogue@gmail.com>
* function to emit metrics * add metrics docs --------- Signed-off-by: technillogue <technillogue@gmail.com>
* function to emit metrics * add metrics docs --------- Signed-off-by: technillogue <technillogue@gmail.com>
* function to emit metrics * add metrics docs --------- Signed-off-by: technillogue <technillogue@gmail.com>
* function to emit metrics * add metrics docs --------- Signed-off-by: technillogue <technillogue@gmail.com>
* function to emit metrics * add metrics docs --------- Signed-off-by: technillogue <technillogue@gmail.com>
* function to emit metrics * add metrics docs --------- Signed-off-by: technillogue <technillogue@gmail.com>
* function to emit metrics * add metrics docs --------- Signed-off-by: technillogue <technillogue@gmail.com>
we need the model code to emit accurate input token counts using the correct prompt formatting and tokenizer for the model in question. after some internal discussion
from cog import emit_metric
seems similar to the best approach for this. presently these metrics are ignored for non-official models and shouldn't really matter anywhere else, so it shouldn't be a concern with billing. this particular approach is janky but should unblock us for now.usage: